Inspiration

  • Kentucky’s accountability system re-design decision to forgo growth and utilize improvement instead.
  • Countless discussions at TAC meetings where accountability system terminology is used interchangeably and incorrectly (most notably by experts in our field).
  • Numerous presentations at TAC meetings discussing poorly designed quasi-experimental studies using results from state summative assessments.
  • Characterizations of “learning loss” and “recovery” during and after the COVID-19 pandemic.
  • Numerous staff meeting discussions about accountability systems and their design.
  • 2023 Brian Gong Colloquium on accountability system design.

Coherence

  • The term coherence is not frequently used in discussions of accountability systems and when it is, it is often not clearly defined.
  • What is coherence with respect to accountability systems?
  • What is incoherent about current accountability systems?
  • Why is coherence important?
  • How can coherence be instantiated in accountability systems?

Gill’s Framework

  • Brian Gill’s (2022) paper, What should the future of educational accountability look like?, introduces a useful framework for indicators used in accountability systems.
  • Three Core Components of Gill’s Framework
    • Processes: The activities, instructional practices, and organizational elements that schools implement to support student learning.
    • Impacts: The measurable effects of processes on student learning and development, often captured through growth metrics or value-added models.
    • Outcomes: The long-term results of education, such as achievement/proficiency levels, graduation rates, and other indicators of student success.
  • Gill argues that for indicators to drive meaningful improvement, they must be:
    • Valid: Accurately measure what they claim to measure without systematic bias.
    • Reliable: Stable and consistent over time to avoid misdiagnosis or loss of credibility.
    • Robust: Resistant to unintended consequences and manipulation while capturing critical dimensions of educational quality.

Gill’s Infographic

  • Gill’s focus is on the quality of the three components as opposed to how they fit together.1
  • Gill’s ordering of the components from left to right opposite of how they occur temporally: Processes lead to Impacts lead to which lead to Outcomes.
  • Though the quality of the components is important, I argue that the connection between the components is more important.
  • This incoherence is what I intend to address and solve.

Coherent Components

  • This modified infographic correctly orders the Processes, Impacts, and Outcomes components.
  • Growth and Status indicators are indicated to the right of the infographic: Growth is a leading indicator whereas Status is a lagging indicator.
  • Growth is the bridge connecting Processes to Outcomes.

Why Coherence is Critical?

Achievement & Growth/Carlson’s Table

sgpFlow: Connecting Processes to Impact to Outcomes

The Importance of Equity

What is Equity?

Accountability as Statistical Process Control

References

Gill, Brian. 2022. “What Should the Future of Educational Accountability Look Like?” Journal of Policy Analysis and Management 41 (4): 1232–39. https://doi.org/10.1002/pam.22428.